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ABSTRACT 

Objectives: To investigate the feasibility of using 
research papers cited in clinical guidelines as a way to 
track the impact of particular funding streams or 
sources. 

Setting: In recent years, medical research funders 
have made efforts to enhance the understanding of the 
impact of their funded research and to provide 
evidence of the 'value' of investments in particular 
areas of research. One of the most challenging areas of 
research evaluation is around impact on policy and 
practice. In the UK, the National Institute of Health and 
Clinical Excellence (NICE) provide clinical guidelines, 
which bring together current high-quality evidence on 
the diagnosis and treatment of clinical problems. 
Research referenced in these guidelines is an 
indication of its potential to have real impact on health 
policy and practice. 

Design: This study is based on analysis of the 
authorship and funding attribution of research cited in 
two NICE clinical guidelines: dementia and chronic 
obstructive pulmonary disease. 
Results: Analysis identified that around a third of papers 
cited in the two NICE guidelines had at least one author 
based in the UK. In both cases, about half of these UK 
attributed papers contained acknowledgements which 
allowed the source of funding for the research to be 
identified. The research cited in these guidelines was 
found to have been supported by a diverse set of 
funders from different sectors. The study also 
investigated the contribution of research groups based 
in universities, industry and the public sector. 
Conclusions: The study found that there is great 
potential for guidelines to be used as sources of 
information on the quality of the research used in their 
development and that it is possible to track the source 
of the funding of the research. The challenge is in 
harnessing the relevant information to track this in an 
efficient way. 
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INTRODUCTION 

Medical research has advanced rapidly in 
recent times in all areas from basic sciences 
(ie, decoding the human genome) to the 



ARTICLE SUMMARY 



Article focus 

■ Explore the feasibility of extracting the funding 
source of research papers cited in clinical 
guidelines. 

■ Identify who funded the research which supports 
the guideline development. 

■ Investigate the shared characteristics of the 
publications cited in the two guidelines chosen 
for the study (dementia and chronic obstructive 
pulmonary disease). 



Key messages 

■ Looking at citation in clinical guidelines could 
potentially be used as one of the tools in the 
impact evaluators toolkit. 

■ For this methodology to be fully exploited, the 
accessibility of data and acknowledgement of 
funders in articles need to be improved. 

■ This study helps to investigate the connection 
between funding inputs with changes in medical 
practice and the pathways for these to arise. 

Strengths and limitations of this study 

■ An assumption is made that the fact that 
a research article is cited in a clinical guideline 
is a proxy for potential impact; however, this 
impact is not proven in this study. 

■ The methodology relies on the correct and full 
attribution of funders in research articles; 
however, evidence shows that this information 
can be incomplete. 

■ NICE guidelines are very closely linked with the 
National Health Service in the UK, as such this 
limits actual results of the study, but the 
methodology could be applied to other similar 
documents. 



development of more precise diagnostic tools 
and novel treatments. At the same time, 
public interest in biomedical advances and 
the appetite for more effective treatments 1 are 
increasing in parallel. The demand for the 
results of biomedical research to lead to 
improvement in healthcare has never been 



higher. 
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Across the research world, and particularly in the 
biosciences, there has been a drive to better demonstrate 
and understand the impacts of research, essentially so 
that funds can be allocated to maximum effect. 1 There 
remains a concern that the research community as 
a whole could be better at translating the findings of 
medical research into tangible health and healthcare 
benefits. 5 7 Thus, the need to better understand 
research impact and in particular the pathways to that 
impact is a key priority for research hinders. 8 11 
However, determining the impact of research is chal- 
lenging, particularly in basic and fundamental research 
where the time lag between original research and 
subsequent impacts on health can be long and the 
attribution difficult to track. In addition, perhaps one of 
the most challenging areas has been trying to under- 
stand the nature of the pathways to and subsequent 
impact of research on policy and practice. 

National and international clinical guidelines are 
intended to bring together the best and most current 
evidence about the prevention, diagnosis, prognosis 
and therapy of clinical problems. Clinical guidelines 
are a form of systematic review and, in the UK, focus on 
the defined medical needs of the National Health 
Service (NHS). It should be noted that clinical guide- 
lines are not standards of care but are recommenda- 
tions to the non-specialist or general practitioner. In 
the UK, clinical guidelines are provided by the National 
Institute for Health and Clinical Excellence (NICE), 12 
and since 2005, these have had legal standing in the 
NHS in England and Wales. As such, the results of 
this particular study are potentially limited to the 
NHS; however, the methodology could possibly be 
applied to guidelines governed by other bodies. The 
guidelines exist to help standardise and improve 
patient care and can help to introduce cost-efficiencies 
to the delivery of healthcare. The guidelines are 
evidence based and their formulation brings pieces of 
important and influential research together. For 
a funder, if research it has supported is referenced as 
part of the evidence supporting a national and/ or 
international clinical guideline, then it is an indication 
that this research is likely to be influencing policy and 
practice. Hence, clinical guidelines are potentially an 

attractive resource to support impact tracking and 

. 13 14 
assessment. 

For those engaged in evaluation, historically it has 
been difficult to extract information from a guideline in 
a way that helps support analysis of the references and 
funding sources: simply put, UK clinical guidelines are 
not designed to support the requirements of hinders 
trying to track the impact of their support. However, 
work is underway at the US National Center for 
Biotechnology Information (part of the National Library 
of Medicine, National Institutes of Health) to digitise the 
content of major international clinical guidelines to 
encourage wider access to their content and enable 
greater ability to mine their content and allow auto- 



mated links to individual cited research papers via 
databases such as PubMed. 

In 2009, the UK Medical Research Council, the Well- 
come Trust and the National Institute for Health 
Research, who among them commit nearly £2bn annu- 
ally to support biomedical and applied health research, 
commissioned a detailed analysis of the research cited 
on a small number of UK clinical guidelines to explore 
the potential of the information in broader research 
impact tracking. The objectives of this research were 
threefold. First, this study explored the feasibility of 
extracting the funding source of the research papers cited 
on a guideline. Second, it identified who funded the 
research cited in the selected clinical guidelines. Third, 
it explored the extent to which there are shared charac- 
teristics of the publications cited in these guidelines. 

The then-current NICE guidelines for the manage- 
ment and treatment of two disease areas were selected: 
dementia (2006) and chronic obstructive pulmonary 
disease (COPD) (2004). 15 16 These guidelines were of 
interest for the purposes of this analysis since (1) they 
had been available unchanged for several years and (2) 
there was a likelihood that all three project sponsor 
hinders would have funded some of the underlying 
research evidenced in the guidelines. The two guidelines 
were also in quite different clinical areas, so the authors 
wanted to see if there were differences in the process 
and/ or adoption of research into practice. For each 
guideline, all cited research was examined to pick out its 
characteristics (eg, age, bibliometric indicators) and 
identify any funding attributions. 

METHODS 
Data extraction 

The hrst step was to extract a list of publications from 
each of the guidelines and export them into a Microsoft 
Excel spreadsheet. This was performed automatically 
using bespoke RAND Europe computer scripts, based on 
the PERL scripting language. Here, we briefly describe 
the methodology since a full description is available 
elsewhere. 17 A total of 744 references were extracted 
from the dementia guideline and 446 from the COPD 
guideline. 

Data cleaning 

The extracted bibliographic references were cleaned 
and structured to permit analyses of funding source and 
paper performance indicators. Any references identified 
as non-academic or peer-reviewed publications (eg, 
references to a website, grey literature) and all publica- 
tions before 1980 were removed since these could not be 
investigated using the Web of Science. 

After extraction and initial cleaning, a total of 616 
references were found for the dementia guideline 
(79.4% of the original 776 references) and 412 refer- 
ences for the COPD guideline (83.9% of the original 491 
references). 
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Data processing 

For the funding analyses, the extracted publications 
were searched for in Web of Science 18 to find the insti- 
tution and country affiliations of the authors listed. One 
aim was to identify the publications with at least one UK 
author on the assumption that this would facilitate 
further funding analysis. Another aim was to use all the 
extracted publications for further bibliometric analysis 
(see Grant et al 19 for a similar methodology) . 

From the dementia guideline, 494 of the 616 extracted 
publications (80.1%) were found in the Web of Science. 
While from the COPD guideline, 335 of 412 publications 
were available (81.3%). Any publications not found in 
the Web of Science were processed individually through 
a search methodology utilising the publication libraries 
of RAND Europe and Cambridge University. All search 
processes were duplicated by a second researcher to 
eliminate errors. 

This methodology used a simple search of both the 
RAND library by article title, using Google Scholar and 
the Cambridge University online library and free access 
journals, through the Google search engine. If the 
author affiliation and country remained unidentified, 
then the RAND library was searched by journal, 
followed by browsing for the article using the reference 
data available. In addition, the title could be searched 
for by keyword within the journal. Finally, where 
possible, Cambridge University print holdings were 
searched to find any articles that were not accessible 
online. 

FUNDER ACKNOWLEDGEMENT 

Where publications had at least partial UK attribution, 
the funding source was searched for in the Research 
Outputs Database. 20 21 This is a database housed at the 
Wellcome Trust recording the funding sources for UK 
and Irish publications in the biomedical sciences for the 
period 1988—2001. Funding acknowledgements were 
found for around one-third of publications using this 
database. For the remaining publications, which fell 
outside of the appropriate date range or were not found 
in the Research Outputs Database, the full text of the 
publication was found and funding acknowledgements 
recorded directly, where available. 

Funding source references were then standardised and 
categorised by broad sector. UK funders acknowledged 
on cited papers were categorised into the following 
categories: industry, not-for-profit, hospital trust, govern- 
ment department, government agency (not controlled by 
ministries), local or regional authority, foundation, none 
given and unknown. 

BIBLIOMETRIC AND PAPER CHARACTERISTIC ANALYSIS 

Author affiliations and country locations were identified 
for 595 of the 616 (97%) dementia guideline publica- 
tions and 402 of the 412 (98%) COPD guideline publi- 
cations. These affiliated papers form the basis of the 
following descriptive and bibliometric analysis. 



The citation impact of the publications referenced in 
both clinical guidelines was analysed by sector using the 
concept of citation profiling. This is based on a normal- 
ising technique called ReBased Impact (RBI), which 
takes account of the field in which a paper appears and 
the date since its publication to effectively provide 
a proxy measure for the 'quality' of each paper. The 
world average RBI is 1; the most highly cited articles have 
an RBI >8. 22 

Whole counts were used throughout this analysis; if 
more than one funding source is cited in a publication, 
this was recorded as one publication for each funding 
source. 

RESULTS 

Attribution by funding organisation 

In a large proportion of cited papers, no funding 
acknowledgement was listed. 

Nearly half (104 of the 228; 46%) of publications in 
the dementia guideline with a UK author did not 
acknowledge any funder. Funding information was 
available for 117 papers, with the full text of seven papers 
inaccessible and, hence, missing the funding informa- 
tion. For the 148 publications in the COPD guideline 
with at least one UK author, 60 included no funding 
acknowledgement (41%). Funding information was 
available for 81 publications. The full text of seven 
publications was not accessible and therefore no funding 
data were available. 

Examination of the funding acknowledgements for 
the Medical Research Council, the Department of 
Health (England), the NHS and the Wellcome Trust 
revealed that these funders were overtly linked to only 
a small proportion of papers cited in the guidelines 
(see figure 1 — numbers of publications by funding 
source) . 

Attributions by funding organisation over time 

To determine whether the practice of acknowledgement 
of funding has improved over time, funding acknowl- 
edgement was analysed by year of publication. Although 
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Figure 1 Numbers of publications by funding source. 



Kryl D, Allen L, Dolby K, et al. BMJ Open 2012;2:e000897. doi:1 0.1 1 36/bmjopen-201 2-000897 



3 



Tracking the impact of research on policy and practice 



it appears that more recent publications have more 
complete funding acknowledgements than older ones, 
over the whole period, and for both guidelines, there 
was no clear statistical relationship between the age of 
publication and the presence or absence of a funding 
acknowledgement (see figure 2 — funding acknowledge- 
ment by year) . 

The clinical guidelines, on the whole, cited recent 
research; the majority of research papers cited in these 
two UK clinical guidelines were published after 2000, 
that is, within 5 years of the release of these guidelines. 
The average duration between the publication date of 
papers cited and the publication date of the citing 
guideline was 5 years for dementia and 3 years for 
COPD. 

Attribution by funding sector 

Industry was not as prominent a funding source for 
publications cited in the dementia guideline, where 
acknowledgements were distributed across a range of 
funding sources across sectors. In the COPD guideline, 
industry was the most frequently linked funder after 
'none given'. 

Attribution by country 

Of the 616 publications extracted from the dementia 
guideline, 228 (37.2%) had at least one UK-based 
author, while from the COPD guideline, 148 publica- 
tions (35.9%) had at least one UK-based author. 



Researchers based in the UK and USA combined were 
linked to the majority of papers cited in both guidelines. 

Despite dominance of the UK and USA-based 
researchers in the cited papers, many other countries 
contributed to these publications. Papers cited in the 
dementia and COPD guidelines were linked to authors 
from 37 and 36 countries, respectively (see figure 3 — 
contribution of the most active countries). 

Attribution by research sector 

The three research hinders use different means of 
disbursing their money. Funding includes grants to 
universities and hospitals, alongside direct support for 
intramural research. Publication analysis by associated 
institutions revealed that researchers with university 
addresses, followed by those with hospital addresses, 
were linked to the bulk of papers cited on both guide- 
lines. More than 80% of publications cited in the two 
guidelines involved authors based at universities. The 
scientific contribution from other types of publicly 
funded institutions, as well as from non-profit institu- 
tions, was low. 

Nearly 20% of the publications cited in the COPD 
guideline involved authors from industry — a slightly 
higher proportion than in the Dementia guideline. 

Citation quality 

Clinical guideline drafting committees are obliged to 
base their recommendations on the 'best' research 
available. The citation impact of the publications 



Funding acknowledgement by year (COPD) 
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Figure 2 Funding acknowledgement by year. 



Kryl D, Allen L, Dolby K, et al. BMJ Open 201 2;2:e000897. doi:1 0.1 1 36/bmjopen-201 2-000897 



Tracking the impact of research on policy and practice 



Figure 3 Contribution of the 
most active countries. 
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referenced in both clinical guidelines was analysed by 
sector using the concept of citation profiling. 

Overall, papers cited across both guidelines had high 
RBIs. Papers linked to universities, companies (industry) 
and publicly funded research institutions were particu- 
larly highly cited. At the time of the analysis, for the 
COPD guideline, cited papers linked to publicly funded 
research institutions had RBI>8; for the dementia 
guideline, papers linked to universities, companies and 
publicly funded research institutions all had particularly 
impressive RBIs (see figure 4 — citation score by institu- 
tional sector). 

DISCUSSION 
Funding attribution 

Perhaps the greatest challenge in research impact 
assessment is dealing with attribution. Attribution of 
research outcomes and impacts to a specific funder is 
complex and, in many cases, improbable. This arises 



from most medical research receiving funding from 
multiple sources, involving a host of researchers (often 
across institutions) and being incremental such that 
considerable time elapses between original research and 
impact on health. The tide has turned on this issue and 
funding bodies are increasingly working together to 
identify where their funding has made a difference and 
contributed to an outcome or impact. Exclusively 
'claiming' impact ignores the complexities and reality of 
scientific research and we are more interested in noting 
our contribution alongside others and learning from 
this. 

Nevertheless, there is much we can do to help us 
better understand the connection between funding 
inputs and changes in medical practice. This research 
project was intended to flesh out some of the issues that 
we face in trying to link research funding to research 
output (ie, research papers) and specific outcomes 
(clinical guidelines). 
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As described, there was some variability in the quality of 
acknowledgement information provided on papers. In 
our analysis, we did not explore whether the differences 
in acknowledgement quality and completeness varied 
according to the nature of the paper (ie, underpinning vs 
more applied research). A study of 43 UK clinical 
guidelines (and associated Health Technology Assess- 
ments) related to cancer demonstrated that the number 
of funding sources acknowledged in papers varied with 
the 'basicness' of the publications: the more clinical 
papers have "fewer (funding) sources and the more basic 
papers have more." 13 

An interesting direction for future research work on 
clinical guidelines would be to investigate whether the 
high proportion of cited publications with at least one 
UK author in the UK clinical guidelines could be the 
result of a specific funding strategy. That is, some 
research could be being funded to help create evidence 
for the development of clinical guidelines. This specific 
funding strategy might explain partially why UK-auth- 
ored publications are over-cited in the UK guidelines, 
relative to the share of publications. 

The advice given to funding recipients by UK binders 
regarding how they should be acknowledged in publi- 
cations, and the extent to which these requirements 
have been enforced, has varied significantly over the 
last two decades. While most binders have included a 
requirement for acknowledgement in their terms 
and conditions of award, it has only been since 2008 
that there has been published guidance about a stan- 
dard format. 23 This may explain why industry is fairly 
highly cited across both guidelines as part of 
a researcher's more stringent contractual obligations. 
However, it is worth noting that the extent to which 
researchers are following the standard format is 
unknown. Furthermore, the reality remains that, given 
the incremental nature of much research, it is not easy 
to precisely attribute a publication to its source of 
funding. 

One broader question is how to ensure that research 
information is accessible in order to avoid spending 
funds on research that either cannot be used or may be 
duplication. 

Temporal issues 

We did find a correlation between the publication of the 
clinical guidelines and the dates of the papers cited 
within it. These results corroborate earlier research. 13 
Other bibliometric studies on UK clinical guidelines also 
found that a significant share of publications cited are 
published within the 10 years prior to the release of 
these guidelines. 19 25 

Therefore, this finding was not unexpected, since 
clinical guidelines should be based on new evidence, 
although on occasion the newest evidence may not be 
the best. However, it would be interesting to investigate 
whether the most recent publications cited are the 
outcome of research specifically funded to support the 
development of a clinical guideline, thus potentially 



explaining the peak in publications cited in the two 
guidelines a few years before their release. 

National contributions 

The UK's contribution is high in both guidelines, since 
its world share in medical research measured by its share 
in the total number of publications published in the 
field amounted to approximately 8.6% in 2006. 26 This 
high proportion of papers linked to UK-based authors 
echoes the finding of previous studies. Interestingly, our 
analysis — where around a third of papers are linked to 
UK-based authors — reveals a higher proportion than we 
have seen in other analyses. 

CONCLUSIONS AND RECOMMENDATIONS 

Having greater access to the work cited on clinical 
guidelines would present new opportunities for hinders 
and the research community alike to better understand 
some of the mechanisms that take research from the 
laboratory to the bedside. While this would be but one 
tool in the research impact evaluator's toolkit, it would be 
one that could be relatively easy to harness if both access 
and acknowledgements were improved. As described, 
work is underway between the UK NICE and the National 
Center for Biotechnology Information, and in the future, 
it is envisioned that other guideline providers will make 
their content available in much more structured and 
accessible formats to permit analyses of this nature. 

However, to bring clarity to the tracking of research 
inputs through to published output, the age-old problem 
of ensuring accurate and complete acknowledgement 
information on peer-reviewed published work needs to 
be addressed. This requires a change in the culture of 
how researchers use acknowledgements and greater 
liaison with publishers and information providers who 
could do much to enhance the quality and completeness 
of funding data as a paper is submitted for publication. 
Clarity and perhaps demarcation between the require- 
ments of an 'acknowledgement' section on a paper and 
description of the 'funding' that has supported the work 
would seem an easy step to help remedy this. 27 This is 
particularly pertinent as information providers such as 
Thomson Reuters and Elsevier are now developing 
complex reporting and analytical tools that provide an 
ability to scrutinise the characteristics of published 
work — including who is described as funding the 
work — in new and exciting ways. 

Furthermore, if the methodology of this paper is to be 
generalised both beyond formal guidelines and to other 
healthcare delivery and research output systems and 
metrics, then novel methods of identifying and tracking 
researchers and their outputs, via global identifier 
systems such as that proposed by the ORCID 28 (Open 
Researcher and Contributor ID) initiative, will be 
important. 

Moves to address the definition of an 'acknowledge- 
ment' may also help to address the issue of 'over- 
authorship' and author inflation that has been seen in 
recent years. This is thought to be fuelled in part by the 
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drive towards impact assessment through national 
research allocation formulae, such as the UK Research 
Excellence Framework. 29 Some contributors listed as 
'authors' might be more appropriately 'thanked'. Many 
journals now contain a section that asks for a description 
of 'contributions' — these often do not contain all those 
listed in the author list so there may be a place for 
a more defined 'acknowledgement as thanks' section. 

Funders should also ensure clarity around their 
requirements for a 'funding acknowledgement'. 
Complexities arise, for example, when research takes 
place in buildings funded by a specific donor or when 
a specific piece of research uses a piece of equipment 
funded by a specific donor. And for how long after 
funding has been received by a researcher should they 
continue to provide a funding acknowledgement? 

We found that there is great potential for national and 
international guidelines to be used as sources of infor- 
mation to help further our understanding on the impact 
of research on practice; the challenge is to be able to 
harness that information in an efficient way — so that we 
are able to use this information to feed into future 
research strategy and thereby make the research cycle 
more effective. 
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